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The Reform Support Network, sponsored by the U.S. Department 
of Education, supports the Race to the Top grantees as they 
implement reforms in education policy and practice, learn from 
each other, and build their capacity to sustain these reforms, 
while sharing these promising practices and lessons learned 
with other States attempting to implement similarly bold 
education reform initiatives. 


This publication features links to tools developed by leading 
practitioners and the Reform Support Network in two locations: 
incorporated into the content and organized into a convenient 
tool box at the end of the publication. 
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Introduction 


Student Learning Objectives (SLOs) 
and Quality Control 


The Race to the Top Reform Support Network (RSN) 
designed this toolkit to help States and school 
districts implement SLOs with the highest degree 
of quality. States and districts achieve quality 
implementation when their SLOs set high yet 
attainable expectations for students and teachers, 
and when student success on SLOs predicts student 
success on other independent measures. The toolkit 
contains templates, guidance documents, sample 
SLOs and other tools, which States and school 
districts can select or adapt for their own purposes as 
they implement a system of quality control. 


For many States and school districts, SLOs are 
becoming the solution of choice to the challenge 
of integrating teachers of non-tested grades 

and subjects into evaluation and compensation 
systems that require measures of student growth. 
As part of their evaluation systems, some States are 
requiring teachers of tested grades and subjects 

to develop student learning objectives as well. 
Although specifics of the SLO approach vary across 
jurisdictions, the implementation process is similar 
in most, though not all, States and school districts. 
At the start of the school year, teachers, principals or 
district administrators set one or more learning goals 
for students and identify a measure or measures 

for assessing progress towards them. In most cases, 
the principal, an evaluator or the district approves 
the learning goal(s) and the assessment(s). In many 
schools, teachers later sit down with evaluators for 

a mid-term conference to discuss progress students 
are making toward the learning goals. At the end of 
the year, teachers and their evaluators assess student 
progress, and evaluators determine how well the 
teacher succeeded in achieving it. 


Though this toolkit focuses reader attention on using 
SLOs to integrate teachers of non-tested grades and 
subjects, SLOs fuel reflective classroom practices and 
good instruction for all teachers. Effective teachers 
have always gathered data, set goals based on that 


data and then assessed whether the goals have been 
met. When well implemented by schools or districts, 
the SLO process brings consistency and rigor to this 
sound practice. It has the added value of providing 
school districts with an authentic way to assess the 
impact educators have on student learning. However, 
implementing SLOs effectively requires more 
attention to instructional practice than administering 
a State-standardized assessment to help determine 
the effectiveness of teachers. 


SLOs present States and school districts with 

the challenge of ensuring their quality, rigor and 
comparability across classrooms, districts and entire 
States. Yet States and school districts cannot expect 
their SLOs to yield the same sort of scientific validity 
and reliability that standardized State assessments 
produce. That is simply not possible. However, there 
is strong precedent in other fields for using objective- 
based outcomes. Employers and employees in many 
American industries sit down together annually 

to set objectives and the metrics they will use to 
determine whether they have been met. Employers 
make decisions about their employees—whether 

to sign them up for training or to advance them, for 
instance—based on the results of the objectives. And 
they do so without using psychometric methods 

to prove that the metrics are relevant, or that 
expectations have been met. Still, employees should 
expect a fair, rigorous and high-quality process of 
setting objectives and implementing them. 


The Reform Support Network has designed this 
quality control toolkit to help States and school 
districts meet this challenge. 


History of SLOs 


While great teachers have always set and monitored 
learning objectives for students, the idea that these 
objectives should be tied to high-stakes decisions 
emerged in Denver in 1999, when the Denver 
Classroom Teachers Association and Denver Public 
Schools agreed to pilot a performance-pay system 
that required teachers of both tested and non- 
tested grades and subjects to set “student growth 
objectives." The evaluation of this pilot revealed 
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two particularly compelling results: Teachers 

who developed high-quality SLOs produced 
better student-achievement gains, and student 
achievement increased as the length of teacher 
participation in the pilot increased.' To this day, 
Denver teachers who meet their objectives receive 
pay increases. 


Soon after Denver established student growth 
objectives as a measure of teacher effectiveness, the 
Austin Independent School District also began using 
student learning objectives to make compensation 
decisions. No State or district in the early years of 
the 21% century, however, used student learning 
objectives as a formal component of educator 
evaluation systems. 


That changed with the arrival of the U.S. 
Department of Education's Race to the Top grant 
program, with first and second-round winners 
required to implement teacher evaluation 
systems that differentiate effectiveness, by using 
multiple rating categories that take into account 
udent growth as a significant factor. While 
many winning States could point to their growth 
measures for teachers of tested grades and 
subjects, they had little or nothing to measure 
he performance of teachers of non-tested 
grades and subjects (for example, kindergarten, first- 
grade and second-grade teachers; special education, 
music and art teachers; and physical education, 
career, technical, social studies and science teachers). 
Some States decided to use school-wide growth 
results as measures of performance for teachers 

of non-tested grades and subjects while others 
developed end-of-course assessments for nearly 
every grade and discipline. Still many others decided 
to implement student learning objectives. Among 
Race to the Top States, these include Colorado, 
Georgia, Hawaii, Louisiana, Maryland, New York, Ohio, 
Pennsylvania, Rhode Island and Washington, D.C. 


1 Catalyst for Change: Pay For Performance in Denver Final 
Report, CTAC (2004), http://www.ctacusa.com/PDFs/Rpt- 
CatalystChangeExecSumm-2004.pdf 
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Overview of Toolkit Contents 


As part of their participation in the Race to the Top 
grant program, leaders from 10 State departments 

of education joined a Student Learning Objective 

Work Group. Participants identified several objectives 
for the work group, including the creation of an SLO 
quality control framework and a toolkit to inform it. 
Leaders from Georgia, Maryland, New York and Ohio 
volunteered to serve on a subcommittee to oversee the 
development of these products. The first stage of their 
work resulted in the following four-stage framework: 


SLO Quality Control Framework 


Provide tools for developing, 
approving and scoring SLOs 


Ensure continuous improvement by 
refining systems based on feedback 


Procure or create 
assessments 


and holding districts and schools 
accountable for quality 


Train district and school 
staff and administrators 


The second stage of the group's work focused on 
identifying existing potential material for each 
component of the framework. The group also asked 
the RSN to create a small number of tools that the 
field has yet to develop. These, they said, included 
those that would help them monitor the approval 

of SLOs, audit SLO scoring and provide an SLO 
certification process for evaluators. The RSN took 

the group's direction and examined the tools they 
suggested and many others. It created the requested 
tools where it could. The tools the RSN finally selected 
and developed now reside in this toolkit. 


Although a work group of States has reviewed and 
strengthened the new tools, the RSN suggests 

that States and school districts view them as initial 
efforts upon which they can build and improve. In 
most cases, these tools provide potential users with 


guidance: questions to answer before choosing an 
approach, cautions to take into consideration and 
options for action. The approach the RSN took to the 
development of these tools distinguishes them from 
many of the materials in the toolkit—which are ready 
for use in the field immediately—and underlines 

the need for jurisdictions to create even better tools 
based on this guidance. 


We have divided the toolkit into the four components 
of the framework. Some of the components have 
more tools than others. Given the current state of 

the art in certain areas of SLO implementation, other 
components are less robust. 


The RSN is grateful to the innovators who produced 
the materials that now populate the toolkit. They 
provide a foundation upon which States can build 
a system of quality control so that teachers of non- 
tested grades and subjects can participate with 
greater rigor and quality in new pay and evaluation 
systems that require measures of student growth. 
While the tools herein represent the current state 
of the art, we should expect SEAs and LEAs to 
improve them over time. In the spirit of continuous 
improvement, the RSN will continue to update the 
toolkit as the field gains greater insight and develops 
even stronger approaches. 


Component #1: 

Provide Guidance, Templates 
and Tools to Develop, 
Approve and Score SLOs 


Component #1: 
Provide Tools 
for developing, 
approving and 
scoring SLOs 


Component #4 Component #2 


Component #3 


The SLO process typically begins with the 
examination of student achievement data followed 
by the development of the SLO itself, its approval, 
monitoring and finally its scoring for inclusion as a 
factor in a teacher's summative evaluation rating 
and/or as a data point for a decision about her future 
compensation. Each step in the process requires 
thoughtful communication and collaboration 
between teachers and their evaluators, a solid 
understanding of the expectations for SLO quality 
and solid tools to support implementation. 


This section of the SLO Quality Control Toolkit provides 
sample approaches to creating, approving, and scoring 
student learning objectives as well as overall guidance 
to those who would implement them. 


Creating the SLO: 
Development Templates 


Teachers and administrators use SLO development 
templates to create and record SLOs for a defined 
interval of instruction. Templates typically outline 
the components of SLOs that teachers and 
administrators must address in their production. 
State and district templates vary in the number and 
types of components they require, though they are 
almost always aligned to the specifics of their SLO 
framework. These components most commonly 
include the targeted student population, baseline 
information on student learning, learning targets, 
district or State standards aligned to those goals and 
a rationale for selection of the target. Sometimes 
templates require teachers to enumerate the 
instructional strategies they will use to meet 

their objectives. 


Early adopters such as Denver Public Schools 
pioneered the basic template that many States have 
modeled their own templates after, used as a starting 
point for modification or, in some cases, altered 
significantly. In one recent and important innovation, 
some jurisdictions such as New York, Louisiana and 
Indiana (p. 37-44) have started to incorporate the 
final scoring mechanics of SLOs into the template 
itself so teachers and evaluators know from the start 
what results they will have to produce to receive a 
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specific rating. New York requires all teachers and 
principals to use a State-developed template that 
includes the same basic elements of early adopters 
but adds to it the State's effectiveness-scoring scale 
so teachers know as they begin SLO implementation 
how evaluators will determine the range of student 
performance that will define their SLO rating. 
Louisiana includes a “scoring plan" in its template so 
that teachers and evaluators establish at the outset 
how a rating will be assigned at the end of the year. 
Indiana does the same. What's more, designers built 
the template around a five-step process that begins 
with the selection of an assessment—noteworthy 
as a Starting point in and of itself—and concludes 
with an end-of-course judgment about the teacher's 
performance. Although most SLO systems begin 
quality control when evaluators apply rubrics or 
checklists to gauge the quality of an SLO, Indiana 
builds a system of quality control into the template 
itself. 


Approving the SLO: 
Approval Rubrics or Checklists 


Once written, an SLO usually goes through an 
approval process informed by a rubric or checklist, 
created by the State or district and designed to 
measure the quality of the SLO and/or determine 

if it meets approval standards. Most rubrics in the 
field are in the tradition of those developed by early 
adopters. They have three to four performance levels 
(for example, unsatisfactory, needs improvement, 
acceptable, excellent) that evaluators use to rate 

the SLO in each of several domains, usually the 
components of an SLO established by State or district 
templates/frameworks. 


Helpful examples of these rubrics come from Denver 
and Austin. In Denver, teachers and principals rate 
SLOs in several areas: rationale, population, interval 
of instruction, assessment, expected gain, learning 
content and strategies. Some consider extensive 
rubrics like Denver's to be teaching tools for those 
learning how to write and score SLOs, as they use 
them to think through how to move SLOs from 

poor to acceptable to great. The number of decision 
points required by these rubrics, however, has 
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resulted in debate within the SLO community about 
whether they make SLO implementation difficult for 
administrators to manage, especially in large schools. 
Furthermore, school districts typically do not consider 
approval ratings (that is, whether an SLO is at an 
“excellent” or “acceptable” level) as part of a teacher's 
final score, so the value of the rating can be unclear. 


As more jurisdictions are implementing SLOs, gaining 
experience and learning from their peers, they have 
begun experimenting with easier-to-use tools to 
evaluate the quality of SLOs and ultimately approve 
them. For instance, Rhode Island initially used a 
comprehensive rubric with several domains and 
performance levels. Rhode Island now asks teachers 
and administrators to consider simply whether an 
SLO is acceptable or unacceptable in three different 
areas. The development of more simplified tools 

for SLO approval is in fact an emerging trend, with 
Indiana, Ohio, and Louisiana (p. 13) using tools that 
function not as rubrics but as checklists aligned to 
standards for high-quality SLOs. The Reform Support 
Network itself, through the leadership of Georgia, 
Maryland, New York, and Rhode Island, has developed 
a draft approval tool in the same tradition. 


Scoring the SLO: 
Scoring Guidance 


District evaluators use scoring guidance and rubrics 
to measure how teachers have performed in pursuit 
of their SLO targets. Scoring criteria are typically 
locally controlled, with schools or districts defining 
success—though many simply implement the 
guidance provided by States. 


Almost all districts and States differentiate teacher 
performance on SLOs by establishing four or five 
performance levels. Some districts and States define 
success in terms of whether targets are “partially 
met;"fully met,’ or “exceeded” Rhode Island uses 
different terms: “Minimal)’"Partial)’’Full’ or “Exceptional 
Attainment.’ Others, such as New York, describe success 
in alignment with its overall evaluation framework's 
effectiveness performance levels (for example, 
ineffective, developing, effective, or highly effective). 


For the purpose of quality control, however, States 
and school districts must decide how much evaluator 
judgment plays a role in determining whether 
teachers meet or exceed SLO targets—and what 
performance level they achieve as a result. Some 
approaches incorporate mathematical formulas and 
computations to promote consistent scoring. Ohio, 
for example, recommends that districts use an SLO 
scoring template to input and calculate individual 
student baseline, post-test and growth scores. 
Teachers then use a scoring matrix to rate individual 
SLOs based on the percentage of students that meet 
or exceed their growth target. 


Similarly, Indiana requires teachers and evaluators 

to set mastery goals for whole classes of students 
and then measure teacher effectiveness against the 
percentage of students who perform within a specific 
numeric range. For instance, a scoring example in an 
Indiana SLO guidebook labels a world history teacher 
highly effective if at least 90 percent of her students 
achieve 85/100 or better on the end-of-course 

world history assessment. For an effective teacher, 
that percentage is 74 percent. For an ineffective 
teacher it is below 54 percent. Many States adhere 

to an approach that utilizes percentages, including 
Louisiana and Georgia. 


A different approach allows evaluators to examine 
the performance of teachers against their objectives 
more holistically. Rhode Island allows teachers 

and evaluators to use multiple measures to 
determine if objectives have been met, requiring 
evaluators to examine a body of evidence and make 
determinations based on a reasoned review of it. 


Some States take aspects of both approaches. 
While Indiana (p. 18-21) requires strict reliance on 
numeric measures for whole-class objectives, it has 
less strict requirements for SLOs written for targeted 
populations. In fact, for targeted-population SLOs in 
Indiana, “evaluators decide which performance level 
best describes the effect the teacher had on his or 
her students’ learning,’ the Indiana SLO guidebook 
RISE suggests, “This decision requires professional 
judgment? 


New York draws on both approaches as well. For 
some teachers and principals who teach grades or 
subjects that have a State-provided growth measure, 
the State will provide the final score for the SLO. For 
other teachers and principals, New York leaves the 
decision on how to score SLOs up to superintendents 
of districts, though it requires teachers and principals 
with multiple SLOs to weight the results of the SLOs 
proportionately based on the numbers of students in 
each SLO. 


Putting Templates, the Approval 
Process and Scoring all Together 


Putting templates, approval processes and scoring 
guidance together, many jurisdictions, such as 
Houston and the Washington D.C. Public Schools 
(DCPS), have created Web-based systems for 
managing each of these steps and tracking all SLOs 
in agiven year. The templates and approval rubrics 
and checklists exist as online forms and functions. 
Districts require teachers and evaluators to enter, 
approve, and score SLOs at the appropriate time 
during the school year. These systems support 
quality control by promoting good practice, making 
the SLO process manageable and making data 
readily accessible for purposes of monitoring and 
continuous improvement. 


Providing Overall Guidance: Sample 
Guidance Manuals 


States and school districts implementing SLOs often 
produce SLO handbooks, soup-to-nuts guides to 

how to implement the SLO process. They explain to 
teachers and evaluators how SLOs fit into the larger 
framework of teacher evaluation, how to gather 
baseline data, set objectives, choose assessments, track 
progress and measure results. The source of many of 
the materials for this toolkit, they often contain sample 
SLO templates, scoring frameworks, SLO exemplars, 
assessment approval checklists and frequently asked 
questions, among other helpful tools. They seek to 
ensure consistency and quality across school districts, 
so that they are implementing SLOs in similar ways. 
Sampled here are leading examples from Austin, 
Indiana, New York and Ohio. 
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Component #2: 
Procure or Create 
Assessments and 
Set Growth and 
Mastery Targets 


Component #1 


Component #2: 
Procure or Create 
Assessments 


Component #4 


Component #3 


While SLOs address the policy challenge of having 
student-based performance measures available for 
teachers in non-tested grades and subjects, they do 
not solve the problem of having instruments of high 
quality to measure student learning. Thus, States 
and school districts must subject the assessments 
used for SLOs to a rigorous evaluation process, while 
taking into consideration timeliness, cost, available 
expertise, resources and quality. Those who use 
assessments need guidance, therefore, as they either 
procure existing or develop new assessments to 
support SLO implementation. 


Guidance for the Selection or 
Development of High-Quality 
Assessments 


States and districts encourage the use of high-quality 
assessments in two ways: offering lists of pre-approved 
assessments or providing criteria and guidance for 
selecting or in some cases designing them. 


The number of and criteria for pre-approved 
assessments varies by jurisdiction. Louisiana provides 
a list and rank of assessments by tier, identifying tier- 
one assessments as those produced by the Bayou 
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State, another State or companies that develop 
assessments for national consumption; tier two as 
credible assessments aligned to State standards; 

and tier three as teacher-developed assessments or 
those that use indirect measures of student learning. 
Louisiana recommends that teachers use tier-one 
assessments when available. 


New York as well provides a list of pre-approved, third- 
party assessments. The State requires districts who wish 
to use a third-party assessment to choose one from 
the list for applicable teachers or principals. New York 
also allows districts, regions and Boards of Cooperative 
Educational Services (BOCES) to develop their own 
assessments for the purposes of SLOs. However, the 
superintendent must ensure that the assessment 

is rigorous and comparable across classrooms, in 
accordance with the commissioner's regulations. 


State guidance for assessment development 
commonly asks implementers of SLOs to examine the 
validity and reliability of assessments in measuring 
student mastery, their alignment to district or State 
standards, the procedures used in administering 
them and how their results are reported. Rhode 
Island provides a worksheet that teachers or LEAs 
can complete to justify the use of an existing or 
teacher-created assessment. Georgia (p. 51-57) has 
created its own strong guidance for school districts. 
It requires them to apply a table of specifications and 
an SLO measure-criteria table to assessments they 
create for SLOs (districts use the same devices to 
select pre-existing measures). 


At the district level, Austin is a leader in providing 
guidance. For their SLOs, Austin educators choose 
from six pre-approved district assessments or 
create an assessment following a set of guidelines 
provided by the district. Teachers must create these 
assessments in collaboration with other educators, 
ensure that they are aligned to State or national 
standards and cover all the content in the learning 
objective. The district also includes minimum 
requirements for rigor and item type to address 
concerns about validity and reliability. 


Setting Appropriate Student Learning 
Targets or Objectives 


Setting the right targets is one of the most challenging 
parts of the SLO process. Some States and school 
districts provide very specific guidance on how to set 
goals while others allow greater latitude. The challenge 
of setting targets is that teachers and administrators 
must give careful thought to whether they have set an 
objective that is rigorous—yet realistic. 


Early adopters sought to achieve rigor and realism 

by asking teachers to set growth, mastery targets 

or both by requiring a specified percentage of their 
students to meet them. Usually, under this approach, 
an SLO expects 75 percent to 85 percent of students 
(and sometimes only those who attend class a certain 
percentage of time) to reach the target. 


As the field of SLO implementers expands and 
jurisdictions address the challenge of setting rigorous 
yet reasonable goals, targets are becoming inextricably 
linked to the selected assessments—and standards 
for setting targets are becoming more nuanced. To 
promote fairness and consistency across classes, 

some jurisdictions mandate the target when they 

use standardized tests for particular subjects and 
grades, and provide detailed guidance when non- 
standardized tests are used. New York (p. 15-18) 
requires the use of specified targets and effectiveness 
ratings for SLOs in subjects and grades with State- 
provided growth measures. For all other courses, 
including those with State assessments that do not 
have State-provided growth measures, New York 
allows districts to determine their own targets and 
scoring processes. The State encourages districts to set 
expectations for growth based on district expectations 
and to use as much historical and baseline data as 
possible to justify student-growth goals. 


Promoting a nuanced approach, Indiana (p. 16- 
25) requires educators to establish targets that are 


tiered or differentiated based on different groupings 
of students’ starting points. The process includes 
setting the target “mastery” of a subject (that is, 
scoring 85 points out of 100 on an end of course 
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assessment) and asking evaluators and teachers to 
collect baseline data, assess student starting points 
and place them into preparedness categories (low, 
medium and high). Then, with guidance provided 
by the State based on the number of students in 
each preparedness level, the teacher works with 
her evaluator to determine the overall percentage 
of students expected to perform at the mastery 
level. For example, a “Highly Effective teacher should 
have all students in the high and medium levels of 
preparedness and most of the students in the low 
level of preparedness achieve content mastery.’ 
Rhode Island also allows for tiered targets. 


At the district level, Houston provides detailed 
direction for both assessments and targets. To 
maximize fairness and comparability, the district 

has assigned teachers in many grade levels and 
disciplines assessment and target types (that is, 
comparative growth, comparative progress and 
attainment) based on their grade and subject. Where 
possible, the district uses district-wide assessments 
and growth or progress oriented targets as opposed 
to attainment. 


DCPS also provides comprehensive guidance that 
supports teachers in writing their targets or goals 

for specific assessments. The guidance includes 
suggested assessments, student performance targets 
in each of the teacher effectiveness levels, and how 
to weight the different components. To promote the 
adoption of this guidance and the resulting quality of 
SLO targets, DCPS prepopulates its aforementioned 
Web-based tool with the relevant assessment and 
target guidance for each teacher. Though teachers 
are free to edit and adapt the prepopulated fields, 
this strategy has vastly increased the quality of targets 
submitted from previous years when guidance was 
not prepopulated. 
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Component #3: 
Train District Staff and 
Administration 


Component #1 


Component #4 


Component #2 


Component #3: 
Train District Staff 
and Administration 


SLOs can pose practical challenges but also can 
transform school and district cultures. By gathering 
data and identifying expected student outcomes 
for every student, teachers and school leaders focus 
on student learning in ways that can drive strong 
instructional practice. The SLO community should 
not underestimate the preparation required to 
successfully implement SLOs. Teachers, principals 
and evaluators need training and supporting tools 
in all stages of the SLO process—from development 
to scoring—to ensure high-quality implementation. 
States and districts are now employing several tactics 
and tools to train their staff. 


Training Modules 


Districts and States have developed various training 
tools to help stakeholders understand the different 
stages of SLO development and implementation. 

In addition to traditional handbooks and in-person 
workshops, some States and LEAs have produced 
training modules in webinar and video format. 
Rhode Island, for example, offers an hour-long, 
comprehensive SLO writing webinar that it makes 
available as a video download through a State portal. 
Rhode Island has also developed a PowerPoint 
training module intended to supplement instructor- 
led, in-person training with other evaluators, 
complete with learning exercises and discussion 
questions. Georgia and New York provide training 
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modules in PowerPoint format as well. New York 
provides PowerPoint modules for superintendents 
and district leadership and supplements them with 
short, focused videos on the SLO process for teachers 
and principals, how to develop SLOs for specific 
grades and subjects, and State resources. 


Denver and Austin offer user-friendly accessible 
videos that stream directly from district websites. 
They are short, focused videos that are less than ten 
minutes each. These types of training modules add 
meaning to and help teachers and administrators use 
templates and other resources available to support 
SLO implementation. 


School-Based Training 


A few districts have invested in school-based SLO 
trainers or “champions” to support implementation. 
Both Austin and Houston, for example, pay teachers 
a small stipend to take on leadership roles in their 
schools. These trainers and champions come 
together several times a year, receive training and 
troubleshoot common challenges. Their districts give 
them training and facilitation materials to conduct 
“turnkey” trainings in their own buildings. At the State 
level, Rhode Island trains a cohort of 20 “intermediate 
service providers” whom Rhode Island trains and 
then assigns to districts to support teachers and 
administrators through the entire evaluation process. 


Sample Annotated SLOs for Training 


All States and school districts implementing SLOs 

have developed SLO exemplars across grade levels 
and subject areas. The exemplars are used in guidance 
documents to give teachers and evaluators specific 
illustrations of what quality SLOs look like. To build on 
the successful development of these models, however, 
the RSN plans to increase the utility of SLO models 

as teaching tools by working with States to annotate 
both high-quality and lower quality SLOs with 
explanatory text. These annotated SLOs will contain 
detailed analyses of what makes specific components 
of SLOs strong or weak. Interested parties will be able 
to view these annotated SLOs in an online library to be 
launched in January 2013. 
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SLO Certification: 
Guidance Provided by the Reform 
Support Network 


While it is common for new evaluation systems to 
require evaluators to become certified observers of 
professional practice, certification for implementation 
of student learning objectives is less common. Some 
jurisdictions require certification for all components of 
the evaluation framework, including student learning 
objectives. Houston created a certification process 
for evaluators that it holds prior to the start of each 
school year. Evaluators attend three days of training 
on the full evaluation framework, including about 

a day covering the SLO process. Training consists of 
presentations, interactive work and quizzes to check 
for understanding. To be ‘certified’ evaluators have to 
attend the training and pass the quizzes. 


As certification is new to many jurisdictions, the RSN 
has developed preliminary guidance for jurisdictions 
considering certification. 


SLO certification over and above extensive 
training offers jurisdictions the following 
advantages: 


1. It creates an explicit mechanism to ensure all 
evaluators have completed their SLO training. 


SLOs are complex, high stakes and new to many 
educators. If a district or State wants to ensure its 
evaluators have participated in and demonstrated 
mastery of a required training sequence for SLOs, 
it can track successful training completion by 
attaching it to certification. 


2. It raises the quality of SLOs by requiring 
evaluators to demonstrate competence in key 
SLO approval skills. 


Implementers of SLOs know that SLO quality 
varies widely. Certification can be one of several 
strategies to raise SLO quality by ensuring that 
evaluators have mastered key characteristics of 
excellent SLOs, along with strategies to support 
teachers to write them. 
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3. It increases the inter-rater consistency of 
SLO scoring. 


Optimally, any evaluator would score an SLO 
identically. Consistency of scoring assures fairness 
for teachers and also provides clarity for teachers 
about what the system values in student learning 
growth. Certification can include scoring practice 
that results in better consistency among evaluators. 
The Reform Support Network has designed a 
protocol for the training of evaluators to ensure 
consistency of quality across SLOs, so States and 
districts can in fact use them as part of certification. 


If a district determines that it would like to 
initiate a SLO certification, it might want to 
consider the following questions: 


1. Will SLO certification be part of the overall 
evaluator certification or a stand-alone 
requirement? 


If the jurisdiction also requires observer certification, 
it can choose either to (a) create an overall evaluator 
certification requirement that has two parts, both 

of which must be successfully completed to attain 
certification, or (1b) let each component stay separate, 
which introduces the possibility that some evaluators 
may be observation-certified but not SLO-certified, 
or vice versa. 


2. Will the jurisdiction grant a certificate solely 
on the basis of attendance, or will it require 
demonstration of SLO administration skills? 


If the jurisdiction's priority is simply to ensure 

and track whether evaluators have completed 
required training, then attendance-based (“sit 
and get”) certification is appropriate. If the 
jurisdiction wants to use the certification process 
to exclude evaluators who do not understand 
SLOs or who bring a point of view about scoring 
that differs from the system's intent, they can 
build demonstrations or assessments into the 
certification requirements. Such demonstrations 
could include, for example, the approval and 
scoring of sample SLOs and the conduct of mock 
approval, mid-year and scoring meetings with 
teachers. Jurisdictions can administer the approval 
and scoring assessments using online modules. 
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3. What will be the pre-practice certification 
requirements, and what will be the requirements 
to maintain certification? 


Evaluator behavior can change over time and so 
the Reform Support Network recommends that 
certification should not be granted for life. In 
addition to the requirements for earning an SLO 
certificate, jurisdictions might specify duration of 
certification and renewal requirements. 


4. What will be the consequence for non- 
certification? 


Certification tests can result in pass/fail 
designations or more differentiated categories— 
that is, “conditionally certified”—to identify staff 
members who can evaluate SLOs with additional 
supervision and need extra support to reach full 
certification. If a jurisdiction chooses to require 
SLO certification, then it will have to commit itself 
to enforcing the consequences: An evaluator 
who does not achieve the certification cannot 
administer, approve and score SLOs without 
support or supervision. 


Component #4: 

Ensure Continuous 
Improvement by Refining 
Systems Based on Feedback and 
Holding Schools and Districts 
Accountable for Quality 

States and school districts want to continuously 


improve their SLO systems so that students benefit 
from effective instruction. To do so, they need to 


Component #1 


Component #4: 
Ensure continuous improvement 
Component #2 


by refining systems based on 
feedback and holding districts and 
schools accountable for quality 


Component #3 


Reform Support Network 


know how teachers and evaluators are implementing 
SLOs in the field. They need to find out if teachers and 
evaluators are implementing SLOs of consistent high 
quality across jurisdictions and that they are scoring 
results accurately. They need to know this information 
so that they can target training to evaluators who 
may not be implementing the SLO process effectively 
or whose scores may be inflated. They also want this 
information so they can hold districts and schools 
accountable for the quality of SLO implementation. 
Assessing system-wide quality of SLOs and using 
SLOs in principal evaluation help ensure both 
continuous improvement and accountability. 


Assessing System-Wide Quality 


One challenge implementers of SLOs face is the 
prospect of monitoring the quality of approved SLOs 
at a scale that allows for informed judgment and 
inspires continuous improvement. When States and 
districts decide if and how to monitor the quality of 
SLOs that principals and others are approving, there 
are a number of trade-offs they might consider. 
Proscriptive monitoring guidance can lead to more 
compliant SLOs but not necessarily more rigorous 
ones. In addition, many SLOs to date have been 
developed in pilots or low-stakes environments so it 
is unclear how the monitoring process will change 
when States and districts enforce higher stakes. 


In addition to these tensions, several challenges come 
with successful State or district monitoring of SLO 
quality. First, some experts have noted that the SLOs 
used in pilots have not been of uniformly high quality. 
Second, elementary-school SLOs tend to be stronger 
than those at the high school level. Variable quality at 
the secondary level stems in part from the fact that it is 
difficult to find an individual principal (or even a team 
of individuals) with enough content knowledge and 
time to evaluate assessments being used in SLOs for 
teachers of different grades and subjects. 


States and districts should assess their own context 
before deciding how much time, energy and funds to 
invest in a monitoring process. If they decide to move 
forward, they can consider such options as those 
presented in the next section. 
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State Approaches to Monitoring and + Encourage every principal to share one high- 

Continuous Improvement quality and one weaker SLO for a more extensive 
district audit. 

States can pursue several strategies to monitor the 

quality of SLOs that principals are approving: - Encourage principal teams to work together to 
audit SLOs and participate in additional training. 

+ Develop, as most States have, State-level quality 


guidelines or rubrics that all principals must use to Auditing SLO Scores 


approve SLOs. 
With new evaluation systems coming on line in Race 
+ Establish an SLO-monitoring and approval to the Top winners and other States, the stakes for 
committee, as Ohio recommends for all school the results of student learning objectives are and 
districts. will remain high. Districts will factor the results into 
employment and career-milestone decisions they 
- Randomly select a few districts across the State make. In any high-stakes assessment system, it is worth 
and have a State committee study their SLOs. cautioning that States and school districts cannot 
implement SLOs with the same scientific precision of 
- Cross-refer district and school results on SLOs with State-wide standardized tests. However, it is sensible 
both observation ratings and large-scale assessment —_ to collect implementation data to refine the existing 
data, a practice to which Rhode Island adheres. system and prudent to check for errors and guard 
against the possibility of cheating. For both purposes, 
District Approaches to Monitoring and jurisdictions will want to monitor the scoring of SLOs 
Continuous Improvement for system improvement and quality control. 


Districts can pursue a variety of strategies to monitor This section of the toolkit should help jurisdictions 
the quality of SLOs that principals are approving: decide whether to audit SLO scoring and provide 
strategies for how to do so. 
+ Set SLOs district-wide in grades and subjects 
where common assessments exist, providing There are three reasons why jurisdictions will 


waiver or opt-out procedures for situations in want to consider monitoring SLO scoring: 
which a district set goal does not work. 


1. To provide feedback for system improvement 
+ Develop, as most districts have, district-level 
quality guidelines or rubrics that all principals 
must use to approve SLOs 


A monitoring system will provide important 
information about the health of the SLO system. 
It will detect, for example, whether the scoring 
approach is confusing for evaluators; whether 


» Embed the monitoring of approvals in regular the assessments used by districts are authentic 
conversations between principals and their measures of student growth; and whether the 
supervisors, so that all principals review a sample system is generating a range of scores, consistent 
of the approved SLOs with their supervisor and with expectations about the variation in teacher 
answer a set of questions about their quality. contributions to student growth. 


+ Establish an SLO-monitoring and approval 
committee that would study the quality of a few 
SLOs in every school. 


14 Reform Support Network 


2. To check for scoring consistency and accuracy 
across teachers, evaluators, schools and 
even States 


In most jurisdictions where teachers have latitude 
to set their SLOs to suit their particular students 
and subject, it will be difficult to achieve perfect 
scoring reliability for SLOs. However, if multiple 
evaluators from different schools scored the same 
teacher's SLO, the score should be consistent 
across evaluators. Furthermore, most experts 
believe that SLO scores should correlate to some 
degree with observation ratings and value- 
added or growth measures where available. It 

is inevitable, especially in systems that require 
the averaging of several SLO scores or other 
mathematical calculations, evaluators will make 
mistakes. A sound system for monitoring SLO 
scoring, along with good training for evaluators, 
is a key strategy for maximizing scoring reliability 
and accuracy. 


3. To detect and deter cheating 


The focus of an auditing system should be on 
continuous improvement; however some States 
have expressed concerns about the potential 

for cheating. A well-designed approach for 
monitoring scoring will not catch all instances of 
assessment outcome manipulation, but it should 
detect some and will deter others. 


Jurisdictions have in fact deployed strategies to 
discourage SLO cheating. Chief among these are 
collaborative (Rhode Island) or district (Georgia) 
goal-setting and having teachers sign assurances 
that they will not manipulate assessment results 
for their SLOs (Georgia). 


There are at least two main approaches to monitoring 
SLO scoring. The first is to analyze SLO scores across 
the jurisdiction to identify overall patterns, differences 
among schools and within subjects. States or districts 
can also cross-tabulate SLO results with value-added 
scores (when available) and performance-observation 
scores. Administrators should examine outliers more 
closely to understand the basis for the differences. 
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The second approach, which can be used 
concurrently, is to examine a random sampling of 
scored SLOs with their associated evidence to verify 
the score. States and school districts can set an 
annual target of verifying 1 to 5 percent of all scores. 


In all systems, the monitoring process can also 
serve to improve the system. Findings may warrant 
changes in the SLO system, changes in training or 
increased communication about the benefits for 
children of thoughtful and authentic goal setting. 


Using SLOs in Principal Evaluation 


Educator evaluation systems can base a portion 
of a principal's evaluation on how well he or she 
implements the SLO component of a new teacher 
evaluation system and how well the teachers she 
supervises perform. These systems can use both 
implementation and outcome-based metrics for this 
purpose. A note of caution about the use of these 
metrics is necessary, however. While this emphasis 
on SLOs in evaluation can provide an incentive 

for principals to help teachers reach their goals, 
they may also provide an incentive for principals 

to lower expectations or inflate teacher scores to 
improve their own evaluation ratings. To combat 
this potential challenge, school districts and States 
where appropriate can consider requiring someone 
other than the principal to approve or score SLOs, 

a practice in which Ohio engages. Guidance issued 
by Ohio suggests that districts can use an aggregate 
measure of all teachers’ SLOs to evaluate principals. 
If a district chooses this option, however, the 
administrator cannot be the sole approver or scorer 
of teacher SLOs. 


Implementation-Based Metrics 


Implementation metrics assess how effectively a 
principal is implementing the SLO process. When 
using these metrics, school districts can gather data 
through surveys and review randomly sampled SLOs. 
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Indiana has established such metrics and includes 
them in its principal practice rubric. Additional 
metrics can include the percentage of teachers 
who can do the following: 


Demonstrate an understanding of the SLO process 
Complete all components of the SLO process 


Believe their principal has implemented the SLO 
process well (for example, “Do you think your 
principal helped you set the right goals?”) 


Agree that their approved SLO(s) meets district 
expectations 


Believe their interactions with their principal were 
of high quality 


Possible survey questions that districts can 
administer electronically or through other 
means include: 


1. Binary Survey Questions (Yes/No) 


— My principal was able to explain the SLO 
scoring process to me. 


— My principal and | discussed the assessments 
chosen for my SLOs. 


— My principal was able to answer my questions 
about setting student growth goals to my 
satisfaction. 


— My principal discussed strategies with me to 
help achieve my SLO targets. 


— My principal established ways to support my 
efforts to promote student learning. 


2. Scaled Survey questions 


Select your level of agreement with the following 
Statements. 

(Strongly agree, agree, neither agree nor disagree, 
disagree, strongly disagree) 


— My principal provided clear expectations and 
guidance for developing my SLOs. 
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— My principal was able to address my questions 
or concerns as | was developing my SLOs. 


— My principal was helpful in selecting an 
appropriate assessment for my SLOs. 


— My principal is knowledgeable about the SLO 
scoring process. 


— My principal provided the appropriate supports 
to help me achieve my SLO targets. 


Outcomes-Based Metrics 


These metrics assess the outcome of the SLO process; 
including what percentage of teachers meet their 
SLO goals. 


Possible outcome-based metrics include the 
percentage or number of teachers who: 


Meet their SLO goals at the end of the year 
Complete their mid-term conferences 


Have approved SLOs that meet standards for SLO 
proficiency established by State or local authorities 


Whose outcomes match other measures (for 
example, student surveys or observations) 


A number of States have proposed the use of 
outcome measures. In Connecticut, teacher 
effectiveness—as measured by an aggregation 
of teachers’ SLOs—accounts for 5 percent of an 
administrator's evaluation. 


ncluding SLOs in the principal evaluation process 
provides principals and other evaluators with 
information about how to improve their practice. 
t also provides school districts with system-wide 
information about how well SLO implementation 
is going in the field. School districts committed to 
continuous improvement can use this information 
to deliver training where appropriate and make 
adjustments to their SLO systems as necessary. 


Reform Support Network 


Conclusion 


In conclusion, SLOs are an evolving practice that 
offers promise for evaluating the contributions 
teachers make to student learning. This process of 
setting rigorous yet attainable goals, implementing 
assessments aligned to the goals, and monitoring 
and evaluating progress builds strong instruction and 
can be adapted to teachers of all grades and subjects. 
However, implementing SLO systems is complex, and 
it relies on evaluator and teacher judgment. Although 


states and school distri 


cts should not expect their 


SLOs to achieve the same sort of scientific validity 
and reliability that standardized state assessments 
produce, fairness and comparability across teachers 


and schools are critica 
of SLOs. This toolkit de 
tools for ensuring qua 


elements for high-stakes use 
scribes emerging practices and 
ity implementation of SLOs 


which states and distri 
for their own systems. 


cts can adapt and build upon 
In the spirit of continuous 


improvement, the RSN will continue to update the 
toolkit as the field gains greater insight and develops 
even stronger approaches. 
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Introduction 
+ RSN SLO Background 


Component #1: Provide Guidance, Templates and Tools to Develop, Approve and Score SLOs 


SLO Development Templates 
+ Denver SLO Template 
+ New York SLO Template 
+ Louisiana SLO Template 


+ Indiana SLO Template 


Approving the SLO: SLO Approval Rubrics or Checklists 
+ Denver SLO Approval Rubric 
+ Austin SLO Approval Rubric 
+ Rhode Island SLO Approval Checklist 
+ Indiana SLO Approval Checklist 
+ Ohio SLO Approval Checklist 
+ Louisiana SLO Approval Checklist 
+ RSN SLO Approval Checklist 


Scoring the SLO: SLO Scoring Guidance 
+ Rhode Island SLO Scoring Guidance 
+ NewYork SLO Scoring Guidance 
+ Ohio SLO Scoring Guidance 
+ Louisiana SLO Scoring Guidance 
+ Georgia SLO Scoring Guidance 


+ Indiana SLO Scoring Guidance 


Providing Overall Guidance: Sample Guidance Manuals 
+ Austin SLO Guidance Manual 
+ Indiana SLO Guidance Manual 
+ New York SLO Guidance Manual 
+ Ohio SLO Guidance Manual 


Component #2: Procure or Create Assessments and Set Growth and Mastery Targets Using Them 


Guidance for the Selection or Development of High-Quality Assessments 
+ Louisiana SLO Assessment Guidelines 
+ New York Approved SLO Assessment List 
+ Rhode Island SLO Assessment Worksheet 
+ Georgia SLO Assessment Measure-Criteria Table 


+ Austin SLO Assessment Guidelines 
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Setting Appropriate Student Learning Targets or Objectives 
+ NewYork SLO Growth Goal Setting Guidance 
+ Indiana Tiered SLO Target Setting Guidance 
+ Houston Student Progress Goal Setting Guidance 


+ DCSLO Assessment and Target Guidance 
Component #3: Train District 


Training Modules 
+ Rhode Island SLO Writing Webinar 
+ Rhode Island SLO Training Module PowerPoint 
+ Georgia SLO Training Module PowerPoint 
+ NewYork SLO Training Module Website 
+ Denver SLO Training Module Video 
+ Austin SLO Training Module Video 


School-Based Training 


+ Houston SLO Campus Learning Session 


SLO Certification: Guidance Provided by the Reform Support Network 
+ RSN SLO Evaluator Protocol for Establishing Consistent Quality 


Component #4: Ensure Continuous Improvement by Refining Systems Based on Feedback 
and Holding Schools and Districts Accountable for Quality 


Using SLOs in Principal Evaluation: Guidance Provided by the RSN and Supported by State- 
Developed Tools 
+ Ohio Principal Evaluation Guidance 


+ Indiana Principal Practice Rubric 


This publication features information from public and private organizations and links to 
additional information created by those organizations. Inclusion of this information does 
not constitute an endorsement by the U.S. Department of Education of any products or 
services offered or views expressed, nor does the Department of Education control its 
accuracy, relevance, timeliness or completeness. 
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